Potential-Based Shaping and Q-Value Initialization are Equivalent

Shaping has proven to be a powerful but precarious means of improving reinforcement learning performance. Ng, Harada, and Russell (1999) proposed the potential-based shaping algorithm for adding shaping rewards in a way that guarantees the learner will learn optimal behavior. In this note, we prove certain similarities between this shaping algorithm and the initialization step required for several reinforcement learning algorithms. More specifically, we prove that a reinforcement learner with initial Q-values based on the shaping algorithm’s potential function make the same updates throughout learning as a learner receiving potential-based shaping rewards. We further prove that under a broad category of policies, the behavior of these two learners are indistinguishable. The comparison provides intuition on the theoretical properties of the shaping algorithm as well as a suggestion for a simpler method for capturing the algorithm’s benefit. In addition, the equivalence raises previously unaddressed issues concerning the efficiency of learning with potential-based shaping. 1. Potential-Based Shaping Shaping is a common technique for improving learning performance in reinforcement learning tasks. The idea of shaping is to provide the learner with supplemental rewards that encourage progress towards highly rewarding states in the environment. If these shaping rewards are applied arbitrarily, they run the risk of distracting the learner from the intended goals in the environment. In this case, the learner converges on a policy that is optimal in the presence of the shaping rewards, but suboptimal in terms of the original task. Ng, Harada, and Russell (1999) proposed a method for adding shaping rewards in a way that guarantees the optimal policy maintains its optimality. They model a reinforcement learning task as a Markov Decision Process (MDP), where the learner tries to find a policy that maximizes discounted future reward (Sutton & Barto, 1998). They define a potential function Φ(·) over the states. The shaping reward for transitioning from state s to s′ is defined in terms of Φ as: F (s, s′) = γΦ(s′)− Φ(s), where γ is the MDP’s discount rate. This shaping reward is added to the environmental reward for every state transition the learner experiences. The potential function can be viewed as defining a topography over the state space. The shaping reward for transitioning from one state to another is therefore the discounted change in this state potential. Potential-based shaping guarantees that no cycle through a sequence of states yields a net c ©2003 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.